Parameter Optimization Algorithm with Improved Convergence Properties for Adaptive Learning

The error in an artificial neural network is a function of adaptive parameters (weights and biases) that needs to be minimized. Research on adaptive learning usually focuses on gradient algorithms that employ problem–dependent heuristic learning parameters. This fact usually results in a trade–off between the convergence speed and the stability of the learning algorithm. The paper investigates gradient–based adaptive algorithms and discusses their limitations. It then describes a new algorithm that does not need user–defined learning parameters. The convergence properties of this method are discussed from both theoretical and practical perspective. The algorithm has been implemented and tested on real life applications exhibiting improved stability and high performance.